Skip to content

fix(fluentbit): render structuredMetadataKeys of the Loki output as structured_metadata_map_keys - #2150

Open
lbarrosby wants to merge 1 commit into
fluent:masterfrom
lbarrosby:fix-loki-structured-metadata-map-keys
Open

lbarrosby wants to merge 1 commit into
fluent:masterfrom
lbarrosby:fix-loki-structured-metadata-map-keys

Conversation

@lbarrosby

Copy link
Copy Markdown
Contributor

What this PR does / why we need it:

The Fluent Bit Loki output (Loki.Params) renders structuredMetadataKeys as structured_metadata_keys:

if len(l.StructuredMetadataKeys) > 0 {
	kvs.Insert("structured_metadata_keys", strings.Join(l.StructuredMetadataKeys, ","))
}

Fluent Bit has never had that property. out_loki only knows structured_metadata and, since 4.0, structured_metadata_map_keys (plugins/out_loki/loki.c config map, checked up to v5.2.0). Output properties are validated against the config map at init (flb_config_map_properties_check in src/flb_output.c), so a ClusterOutput with structuredMetadataKeys makes Fluent Bit fail with unknown configuration property 'structured_metadata_keys' and the Loki output never starts.

This renders the field as structured_metadata_map_keys, the Fluent Bit option for "record keys whose values become structured metadata": each listed key (a record accessor such as $kubernetes['labels']) must point to a map, and its key/value pairs are added as structured metadata. The field description, the CRDs, manifests/setup/setup.yaml and docs/plugins/fluentbit/output/loki.md are updated to say so; single values keep going through structuredMetadata (key=${record[...]}).

Tests:

  • TestLokiOutputWithStructuredMetadata_Load and _LoadAsYaml expected structured_metadata_keys level,caller; they now use map keys and expect structured_metadata_map_keys. With the old key name in loki_types.go both fail.
  • go test ./apis/... passes, golangci-lint run on the output package reports no issues.

Which issue(s) this PR fixes:

None, found while comparing the keys the operator renders with the Fluent Bit config maps.

Does this PR introduced a user-facing change?

Fix the Fluent Bit Loki output rendering `structuredMetadataKeys` as the unknown `structured_metadata_keys` property; it is now `structured_metadata_map_keys` (Fluent Bit 4.0+).

Additional documentation, usage docs, etc.:

docs/plugins/fluentbit/output/loki.md

…tructured_metadata_map_keys

Fluent Bit has no structured_metadata_keys property; its Loki output
rejects it as an unknown configuration property and the instance fails
to start. The option for record keys whose maps become structured
metadata is structured_metadata_map_keys (Fluent Bit 4.0+).

Signed-off-by: Lucas Barros <lucasbarros.kam@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant